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INTRODUCTION 


Importance  of  Cognitive  Workload  Assessment 

Workload  assessment  is  of  great  interest  to  cognitive  psychologists.  Although  “workload” 
has  not  been  clearly  defined,  it  consists  of  a  set  of  task  demands,  operator  efforts,  and  task 
performance  (Gartner  &  Murphy,  1979).  Cognitive  psychologists  try  to  understand  how  the 
intended  task  demands  are  perceived,  processed,  and  performed  by  an  aircraft  operator. 

Advancement  in  avionic  technology  has  altered  a  great  deal  of  the  traditional  man-machine 
relationship  in  the  cockpit.  The  computer  hardware  and  software  in  the  aircraft  can  assume  control 
and  monitoring  of  a  routine  flight,  handle  repetitive  tasks,  and  automatically  diagnose,  adjust,  and 
correct  mechanical  errors.  This  achievement  in  automation  has  drastically  reduced  a  large  number 
of  psychomotor  or  physical  task  demands  (Hart  &  Sheridan,  1984).  Cognitive  demands,  on  the 
other  hand,  have  become  increasingly  important  for  the  pilot. 

Today,  computer-automation  and  its  display  systems  constitute  the  key  element  of  cockpit 
environment.  The  automated,  sophisticated,  high-speed  man-made  “brain”  constantly  interacts 
with  the  pilot  and  to  receive  commands  under  critical  flight  conditions.  Thus,  piloting  is  no  longer 
a  one-way  communication  from  the  pilot  to  the  instrument.  In  most  cases,  it  is  neither  a  direct  nor 
an  analog  control  of  the  aircraft.  The  versatility  of  the  artificial  brain  provides  the  pilot  with 
instant  messages  for  actions  or  reactions.  It  also  lists  various  options  for  pilots  to  judge,  choose,  or 
reject  in  decision  making.  This  constant  feedforward  and  feedback  between  the  human  brain  and 
the  artificial  brain  inevitably  means  that  cognitive  processes,  such  as  attention,  perception, 
comprehension,  reasoning,  judgment,  memory,  learning,  decision-making,  and  problem-solving, 
are  dominant  features  in  piloting. 

Psychologists  are  increasingly  aware  of  the  types  of  task  demands  imposed  on  pilots. 
Questions  they  are  trying  to  answer  are:  what  cognitive  processes  are  involved  and  what  cognitive 
resources  are  available  for  a  mission  flight?  The  former  is  concerned  with  simulation  of  cognitive 
demands  and  the  latter  involves  resources  identification  and  utilization.  Understanding  the 
underlying  cognitive  processes  of  complex  flight  tasks  would  benefit  pilot  training  by  identifying 
and  structuring  priorities  of  tasks  and  specifying  conditions  under  which  optimal  cognitive 
functions  and  outcomes  could  be  achieved.  Similarly,  knowledge  of  a  pilot’s  resources  available 
for  a  mission  flight  could  assist  aviation  engineers  in  designing  a  friendly  cockpit  environment 
with  a  feasible  flight  model  which  would  best  utilize  human  resources  without  causing  overload 
(performance  deterioration)  or  underload  (boredom).  Finally,  a  comprehensive,  valid  and  efficient 
tool  for  cognitive  evaluation  would  be  useful  for  screening  and  clinical  assessment.  Hence,  the 
importance  of  cognitive  workload  assessment  can  not  be  understated. 

Approaches  to  Workload  Assessment 

According  to  Wierwille,  et  al  (1979),  workload  assessment  techniques  could  be  evaluated 
by  a  matrix  of  two  dimensions:  workload  methodology  and  operator  behavior.  Methods  of 
workload  assessment  include  subjective  opinion  (rating  scale,  interview,  or  questionnaire),  spare 
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mental  capacity  (task  analytic,  secondary-task,  occlusion),  primary  task  measures  (single  or 
multiple  measure,  mathematical  modeling),  and  physiological  measures  (single  or  combined 
measure).  Berliner,  et  al  (1964)  categorized  operator  behavior  into  perceptual  (searching  for  and 
receiving  information,  identifying  objects,  actions,  events),  mediational  (information  processing, 
problem-solving,  and  decision-making),  communication,  and  motor  (simple/discrete, 
complex/continuous). 

Critical  Problems  Associated  with  Assessment  Approaches 

Most  workload  measures,  in  terms  of  measuring  perceptual,  cognitive,  and  communication 
workload,  have  had  limited  success  (Casali  and  Wierwille,  1984;  Hart  and  Sheridan,  1984; 
Hartman  and  McKenzie,  1979;  McKenzie  and  Hartman,  1979;  Rahimi  and  Wierwille,  1982; 
Wierwille,  Williges,  and  Schiflett,  1979).  The  underlying  causes  of  failure  to  validly  assess  or 
predict  a  pilot’s  task  performance  in  reality  can  be  attributed  to:  (a)  the  lacking  of  a  consensus  on 
the  definition  of  workload  and  its  performance  standards  (McKenzie  and  Hartman,  1979);  (b)  no 
theoretical  foundation  for  the  test  development  (Hart  and  Sheridan,  1984);  (c)  issues  with 
sensitivity  and  intrusion  in  simulation  procedures  (Casali  and  Wierwille,  1984;  Heffley,  Clement, 
and  Jewel,  1982);  (d)  inadequate  attention  to  interactions  between  task  loading  and  operator 
factors  (O'Donnell,  1982),  (e)  irrelevance  or  meaningless  in  task  assignment  (Rabbitt,  1984; 
Witlin,  1984);  and  (f)  a  general  dissociation  between  objective  and  subjective  measures  (Gopher 
and  Braune,  1984;  Vidulich  and  Wickins,  1983);  further  reducing  validity,  in  our  opinion,  is  the 
common  use  of  static,  single-mode  tasks  with  no  interactive,  multitask,  dynamic  challenges. 

A  review,  by  this  author,  of  task  descriptions  of  some  latest  assessment  batteries,  such  as 
Criterion  Task  Sets  (CTS),  Unified  Tri-Service  Cognitive  Performance  Assessment  Battery  (TS- 
CPAB),  Basic  Attributes  Tests  (BAT-Version  4),  and  other  similar  instruments,  has  revealed  a 
number  of  psychometric  flaws  for  their  intended  applications.  First,  most  subscales  or  subcategory 
tasks  were  put  together  in  a  piecemeal  fashion  without  clear  rationale  behind  the  arrangement. 
While  each  test  has  an  empirical  literature,  the  “battery”  suffers  from  lack  of  validity  studies.  This 
was  especially  critical  in  dual-task  or  secondary-task  assignments.  Lack  of  Gestalt  in  task 
arrangement  is  distractive  and  could  negatively  affect  the  subject’s  willingness  to  actively 
participate  in  assessment  and  hence  reduce  the  face  validity  of  the  instrument.  Second,  tasks  were 
essentially  measures  of  basic  perceptual  skills  leaving  many  high-level  cognitive  processes 
untouched.  Although  cockpit  facilities  have  been  greatly  reduced  by  introducing  multifunctions 
display  systems  (Statler,  1984),  many  test  developers  remained  committed  to  developing  tasks  of 
monitoring  constancy  or  detecting  changes  on  traditional  panel  displays.  Thus,  many  high-level 
cognitive  tasks,  which  the  automation  is  unable  to  do  (Hart  and  Sheridan,  1984),  have  been 
seriously  ignored.  Finally,  the  pilot’s  knowledge  base,  which  affects  an  individual’s  perception 
and  performance  of  task  demands  (Hart  and  Sheridan,  1984;  Santilli,  1985),  was  omitted  from 
current  measures.  Lack  of,  or  inadequate,  knowledge  of  specific  task  demands  or  unexpected  task 
conditions  could  cause  errors  which  might  either  lead  to  irreversible  accidents  or  become  an 
extraneous  source  of  mental  workload  (Hart  &  Bortolucci,  1984). 
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Principles  of  an  Effective  Cognitive  Workload  Assessment 

Flight  is  a  purposive  behavior.  To  validly  measure  cognitive  workload  in  flight,  the 
principles  listed  below  should  be  followed. 

A  comprehensive  flight  model  should  be  established  to  guide  the  development  of  assessment 
tasks.  A  comprehensive  flight  model  should  clearly  define  workload,  specify  hierarchical 
structures  of  man-machine  relationship,  describe  conditions  under  which  optimal  results  can  be 
reached,  and  identify  criteria  against  which  behavior  outcomes  can  be  evaluated. 

Lower-level  cognitive  processes,  such  as  perceptual  skills  and  attentional  distributions, 
should  be  measured  within  a  contextual  framework  of  higher-level  cognitive  processes,  such 
as  metacognition,  decision-making,  or  problem-solving.  This  will  insure  that  all  aspects  of 
assessment  are  meaningful  and  relevant  to  the  goal  of  flight  and  hence  the  result  can  be  interpreted 
in  a  significant  manner. 

Task  performance  should  reflect  a  human-environment  dynamic  system  of  interactions. 

Since  cognitive  processes  in  flight  are  constantly  modified  by  results  of  interactions  between  the 
pilot  and  his  environment,  simulation  tasks  should  allow  the  subject  to  see  results  of  his  actions 
and  show  records  of  his  inputs. 

Measurement  instruments  should  be  periodically  revised  in  light  of  both  technological 
advancement  and  strategic  innovation.  Recent  development  in  workload  assessment  did  reflect 
some  degrees  of  concerns  for  technological  advancement.  However,  the  strategic  innovation 
which  is  closely  associated  with  the  technological  upgrade  was  very  much  unattended.  What  does 
automation  mean  to  the  pilot?  How-does  the  pilot  perceive  enemy  threats  when  the  integrated 
fire/flight  (IFF)  system  is  on  hand?  Issues  related  to  new  aircombat  strategies  should  be  addressed 
in  the  development  of  assessment  Instruments. 

POPCORN  as  an  Assessment  Tool 

A  Supervisory  Control  Simulation  program  (POPCORN)  was  developed  by  NASA  Ames 
Research  Center  (Hart,  1984).  It  was  designed  to  simulate  the  cockpit  environment  of  the  future. 
The  program,  according  its  developer,  will  measure  such  cognitive  processes  as  attention, 
monitoring,  memory,  rule  learning  and  application,  predicting,  decision  making,  problem-solving, 
and  automated  system  management. 


OBJECTIVE  OF  THE  RESEARCH  EFFORT 

The  objective  of  this  effort  was  to  convert  a  supervisory  control  simulation  system 
developed  by  NASA  Ames  Research  Center  to  a  microcomputer  platform  and  to  conceptually 
analyze  the  POPCORN  as  a  cognitive  workload  assessment  instrument  by  mapping  its  task 
performance  into  cognitive  processes  of  piloting.  This  was  an  initial  step  toward  validation  of  the 
instrument. 
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METHOD 


An  Examination  of  Cognitive  Demands  in  Future  Cockpit  Environment 

An  Aerospace  Medical  Panel  Symposium  was  held  to  address  “Human  Factors 
Considerations  in  High  Performance  Aircraft”  (Hennessy,  1984).  The  symposium  generally 
agreed  that  the  future  aircraft  design  and  air  combat  tactics  would  impose  many  more  cognitive 
demands  than  have  traditional  ones.  The  cognitive  demands  that  were  specifically  mentioned  for 
future  flight  were:  selectivity  of  perception  (attention),  interpretation  (comprehension), 
monitoring  (supervisory  control),  anticipation,  prediction,  recognition  and  diagnosis  of  unexpected 
situations,  learning  and  memory,  inductive  reasoning,  judgment,  rule  application,  risk,  decision 
making,  problem  solving,  and  self-regulation. 

Task  Analysis  of  the  POPCORN 


Basic  Features 

The  POPCORN  consists  of  computer-generated  performance  tasks  to  simulate  cognitive 
processes  of  future  cockpit  environment.  According  to  its  1984  experimental  version,  the 
POPCORN  includes  five  tasks  and  fifteen  functions  for  each  task.  The  five  tasks  are  displayed 
horizontally  at  the  bottom  of  the  cathode-ray  tube  (CRT),  and  the  fifteen  functions  are  displayed 
vertically  in  top-down  sequence  on  the  right  edge  of  the  CRT.  Each  task  is  represented  by  a  box 
which  contains  up  to  20  events  (“kernels”).  When  a  task  (e.g..  Box*)  and  a  function  (e.g.,  “Open” 
a  lid)  are  sequentially  activated  by  a  magnetic  pen  on  a  pad  (with  similar  displays  on  the  CRT), 
kernels  will  start  "popping”  out  of  the  box  and  can  be  “consumed"  (scored)  by  touching  the 
PERFORM  key  at  the  bottom-right  comer:  otherwise,  they  will  “die”  (scored  as  error).  The 
kernels  in  the  activated  box  will  begin  to  mill  around  with  increasing  speed  the  longer  they  remain 
in  the  box  unperformed.  Boxes  can  be  simultaneously  activated  (linked)  and  can  be  stopped  or 
delayed  or  changed.  The  experimenter  can,  by  modifying  task  instructions  or  computer  software, 
change  the  nature,  speed,  meanings,  and  values  of  tasks  and  functions,  and  length  of  trial  to 
increase  the  flexibility,  complexity,  and  demands  of  task  performance. 

Independent  Variables 

1 .  Number  of  tasks  in  action. 

2.  Number  of  functions  used:  OPEN  (box),  CLOSE  (box),  STUPT  (elements),  REMOVE 
(barriers  from  elements),  STOP,  CHANGE  (colors),  ADD  (elements),  SEED 
(elements),  LINK  (tasks),  UNLINK  (tasks),  BACKUP  (for  diagnosis),  CYCLE  (door), 
MANUAL  (for  problem-solving). 

3.  Number  of  touches  of  PERFORMANCE  key. 

4.  Task  difficulty  (Number  of  tasks  X  Number  of  functions). 

5.  Strategies  (combinations  of  tasks  and  functions  in  various  orders  at  different  time). 
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Dependent  variables 

1.  Score  or  penalty  point  (objective  measure). 

2.  Satisfaction  (subjective  measure). 

3.  Stress  (subjective  measure). 

4.  Fatigue  (subjective  measure). 

Mapping  POPCORN  Tasks  into  Cognitive  Processes 

Since  POPCORN  is  a  set  of  dynamic  simulation  tasks,  it  can  evaluate  interactions  among 
tasks,  cognitive  (mediational)  processes,  and  task  performance  outcomes.  The  following 
diagram  depicts  the  dynamic  relationship  among  variables  of  interest. 


To  proceed  with  the  mapping,  each  task  performance  was  conceptually  simulated  and 
sequentially  checked  with  cognitive  processes  to  find  out  the  most  appropriate  matches  in  light  of 
possible  outcomes. 


RESULTS 

Cognitive  Processes  Measured  by  the  POPCORN 


Selective  Attention 

Attending  to  tasks  selected,  procedures  to  be  followed,  and  outcomes  to  be  expected. 
Divided  Attention 

Attending  to  two  or  more  tasks  simultaneously  activated. 

Basic  Perceptual  Skills 

Recognizing  tasks  to  be  completed,  functions  available  for  uses,  and  warnings  of  penalty; 
estimating  speeds  of  popping  kernels  and  possible  scores  can  be  earned;  classifying  tasks  into 
various  classes  (completed  vs.  incomplete,  active  vs.  inactive  or  delayed,  regular  vs.  added, 
linked  vs.  unlinked);  monitoring  progress  of  each  task;  and  predicting  success  against  failure. 

Judgment  and  Decision  Making 

Judging  the  quality  and  quantity  of  strategies  used;  determining  underload,  overload,  or 
just-right  workload;  and  deciding  which  options  to  choose  to  facilitate  scoring,  or  avoid  penalty. 
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Cognitive  Strategies 

Formulating  or  modifying  task  strategies  to  achieve  maximum  scores  with  high  satisfaction 
but  low  stress  and  fatigue. 

Learning 

Acquiring  and  applying  tasks-related  rules;  and  modifying  behavior  through  feedback. 
Memory 

Sensory  store  for  each  popping  kernel;  short-term  memory  for  each  action  taken  or  score 
earned;  and  long-term  memory  of  strategies  used,  rules  followed,  problems  solved,  mistakes 
made,  and  total  scores  accumulated. 

Problem  Solving 

Solving  the  built-in  problems  in  tasks  by  following  predetermined  procedures. 
Metacognition 

Knowing  what  one  wants  to  do,  what  one  has  done,  how  tasks  are  to  be  done,  and  how 
many  mental  resources  are  still  available;  and  knowing  what  one  knows  and  what  one  does  not 
know  in  dealing  with  tasks. 

In  summary,  as  the  result  of  this  conceptual  analysis,  the  POPCORN  appears  to  be  able  to 
meet  the  need  of  assessing  major  cognitive  demands  of  future  cockpit  environment,  although  some 
are  more  adequately  measured  than  others. 

Unique  Characteristics  of  the  POPCORN 

The  impressive  capacity  of  the  POPCORN  to  measure  a  broad  range  of  cognitive  processes 
could  be  attributed  to,  among  other  things,  its  three  unique  features. 

Man-Machine  Interaction 

From  the  beginning  to  the  end,  the  whole  simulation  program  is  a  process  of  action-and- 
interaction  between  the  operator  and  the  task.  Perceptions,  efforts,  and  outcomes  are  changing  as 
time  and  tasks  progress.  It  considers  the  operator  as  an  active,  purposive,  and  adaptive 
information  processing  system  as  did  Newell  and  Simon  (1972).  As  a  result,  all  efforts,  decision¬ 
making,  strategies  and  other  cognitive  processes  are  “naturally”  imbedded  in  the  seemingly  non- 
intrusive,  game-like  task  performance.  Consequently,  human  interest  can  be  maintained;  the 
process  of  task  attainments  can  be  analyzed;  and  the  outcome  can  be  meaningfully  interpreted. 

Metacognition  Assessment 

One  advantage  of  having  a  complete,  meaningful,  and  interactive  simulation  system  is  that 
it  can  be  used  to  measure  the  pilot’s  metacognition.  Metacognition  is  a  superordinate  executive 
system  which  overviews  and  guides  on-going  cognitive  activities  (Meichenbaum,  1977).  For  a 
pilot,  knowing  what  he  knows,  what  he  is  doing,  and  how  things  can  be  done  is  essential  in 
determining  courses  of  action.  Perhaps,  this  tool  can  decide  how  much  a  pilot  will  benefit  from 
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his  own  experience.  Further,  POPCORN  may  provide  salient  information  on  cognitive  efficiency 
or  superior  cognitive  performance  and  functioning. 

Future  Assessment  Tool 

The  POPCORN  was  designed  with  considerations  of  high  degree  of  automation  and  high 
demands  of  cognitive  processes  in  future  aircraft.  In  the  cockpit,  automation  lets  the  pilot  “fall 
behind;”  he  has  to  manage  to  "stay  ahead"  (Richter,  1984).  POPCORN  provides  a  superior  test 
bed  to  understand  cognitive  processing  in  aviation. 


SUMMARY  AND  CONCLUSION 

The  purpose  of  this  project  has  been  to  convert  a  cognitive  workload  task  (POPCORN)  for 
use  with  MS-DOS-based  microcomputers  and  to  evaluate  its  utility  as  a  cognitive  assessment  tool. 
The  programming  has  been  completed  and,  while  certain  cosmetic  improvements  could  be  made, 
it  is  now  a  useable  tool  for  further  research. 

Although  many  current  workload  assessment  instruments  have  had  some  degree  of  success, 
they  are  characterized  by  a  number  of  problems:  a  consensus  on  the  definition  of  workload  is  still 
lacking;  a  comprehensive  theoretical  foundation  for  simulation  is  not  available;  intrusion  effects 
on  measurement  sensitivity  are  unsolved;  man-machine  interactions  are  ignored;  task  assignments 
are  mostly  irrelevant  or  meaningless;  discrete  molecular  tasks  are  put  together  in  a  piecemeal 
manner  to  simulate  the  continuous  and  complex  molar  task  of  flying;  few  high-level  cognitive 
skills  are  being  measured;  and  the  knowledge  base  of  piloting  is  not  properly  included. 

An  analysis  of  the  cognitive  demands  of  future  aircraft  and  combat  strategies  indicates  that 
a  new  workload  measurement  instrument  is  needed  to  adequately  assess  high-level  cognitive 
processes  of  piloting.  Further  analysis  of  a  recently  developed  workload  assessment  tool, 
POPCORN,  shows  that  it  can  monitor  selective  and  divided  attention,  basic  perceptual  skills, 
judging  and  decision-making,  cognitive  strategies,  learning,  memory,  problem  solving  and 
metacognition.  The  study  concluded  that  the  POPCORN  is  an  interactive  simulation  system, 
capable  of  measuring  metacognition,  and  that  it  is  a  unique  example  of  assessment  tools  of  the 
future. 


RECOMMENDATION 

Validity  is  the  major  quality  with  which  a  measurement  can  be  properly  evaluated.  Low 
validity  and  poor  generalizability  of  current  workload  assessment  instruments  have  prompted  two 
possible  options:  continuously  modifying  current  instruments  or  constructing  a  new-generation 
assessment  tool.  The  Basic  Attributes  Tests  (BAT)  and  Unified  Tri-Service  Cognitive 
Performance  Assessment  Battery  (CPAB)  and  other  similar  instruments  are  the  result  of  the  former 
attempt  and  the  POPCORN  the  latter.  On  the  ground  specified  in  the  present  analysis,  POPCORN 
should  be  further  considered  as  a  cognitive  workload  assessment  tool  of  the  future. 
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The  result  of  this  study  suggests  that  the  POPCORN  is  capable  of  measuring  a  large 
number  of  important  cognitive  processes  of  piloting.  This  study  is  only  a  first  step  toward 
validation  of  the  instrument;  a  long-term  empirical  validity  study  should  follow.  To  achieve  this 
goal,  however,  a  short-term  empirical  study  is  recommended  to  systematically  investigate  how  a 
selected  cognitive  process  is  simulated  by  POPCORN  tasks. 
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